An Analysis of Two House Rules in 5E: Disadvantage Shooting into Melee and Flanking

 A game I am currently playing in is using a couple of houserules that Pathfinder players will be familiar with adapted to 5E. Given the nature of these rules, I wanted to investigate how much they affect play, in particular, how much they favor melee over ranged attacks vs base 5E. The rules can be summarized as follows:

• An attacker firing at a target engaged in melee combat (i.e. threatened or threatening a target with a melee weapon) has disadvantage
  
  
• An attacker flanking a target with an ally (i.e. an ally is on the opposite side of the target) gains advantage on their attacks.

These rules should not be confused with existing 5E rules:

• If you're using a ranged attack (Ranged Weapon like a bow, throwing a Melee Weapon with the Thrown ability, using a spell that makes a Ranged Spell Attack), if there is an enemy next to you, you have disadvantage on that attack. Certain feats or class features might negate that penalty, but by default, you the attacker are distracted and suffer disadvantage when there's an enemy next to you while making a ranged attack, whether or not that attack is targeting that close-up enemy.
• This rule can generally be avoided through movement.
  
  
• If you are making an attack, especially a ranged attack or one with a Reach weapon, and you're shooting/reaching past a creature/object/foliage/whatever, your target may have half, 3/4, or total cover against your attack. Half Cover will give them +2 AC, 3/4 Cover will give them +5 AC, and Total Cover will mean you can't even make an attack against them. Ultimately, the DM makes the decision of whether a creature has no cover, half, 3/4, or total... but you should generally expect that shooting through combat might give them at least Half Cover, if your DM remembers to apply the rule. 
• Notice this rule can generally be avoided in melee combat by repositioning as a ranged attacker. 

Let's consider a dex-based fighter with a rapier and a longbow. We will assume the target has AC 15 and that the fighter is 5th level with 18 DEX, +3 proficiency, and 2 attacks.  Normally the fighter can make two attacks with the rapier or two attacks with the longbow:

• For both the rapier and the longbow, 2 attacks, +7 attack vs AC 15, 1d8+4 damage
• Expected damage: 12.35
• Generally a ranged fighter can keep out of melee range, so no disadvantage normally appears in combat.

Now, let's consider with the house rules:

• Assume our melee fighter can gain advantage half the time
• Melee expected damage: 13.57
• Assume our ranged fighter is firing into melee 40% of the time
• Ranged expected damage: 10.29

How big is this difference? Well, consider the ranger's hunter's mark:

• For two attacks, the expected damage from hunters mark in the normal case is 4.55.
  The difference between the ranged fighter and the melee fighter is 3.28 or about the equivalent of 70% of hunters mark or a different equal to about a d4 difference in damage. 

What does this tell us?

• These two house rules effectively favor melee over ranged attacks by about 25% in the case of an equal comparison. 

It should also be noted that this affects all ranged attack spells. When considering spells, this greatly reduces the power of major cantrips such as firebolt (a staple of Eldritch Knight) and Eldritch Blast, a staple of Warlock. It does not affect the rogue's sneak attack, making it more advantageous for classes like melee fighter and barbarian. 

For the warlock case, for 5th level eldritch blast is converts a 3.575 damage beam into effectively a 2.332 damage beam when the target is engaged in melee. With the conservative assumption that this happens 40% of the time, that is an overall weighted reduction to 3.077. It is the equivalent of dropping the 1d10 damage to a 1d8. 

The takeaway from this analysis is that if you make house rules, make sure you understand the impact you are having on the underlying system. If you start with the assumption that ranged and melee attacks are balanced in base 5E, this clearly moves the favor to melee. If you assume that melee had the advantage already, which may be the case with certain optional feats in play, it probably makes that gap a lot wider.