Fencing of solid wood structure is not the best form of barrier next to any planting area, nor playing area for that matter, this is simply because when the wind blows a turbulent down draft is created on the far side of it; this applies to high walls as well.

If it must be used, then an alternative building pattern, using alternate planks on each side to create an open gap in-between, will bring the wind pressure down