Unfortunately Duke men's basketball was too good back after they won two national championships in the early 90's (and that was after those many Final Fours before then). ESPN and Dick Vitale only made them more popular as did their rivalry with North Carolina (1993 national championship and 1991 Final Four). I also remember Michael Jordan in 1982 and Jim Valvano and the miracle NC State upset in 1983. ACC football before Florida State meanwhile was basically irrelevant. The ACC was going to cater to the basketball schools that were making noise on a national level, not the football schools that weren't (or for a long time, football school, Florida State).

If you hate Duke in the ACC or their influence on the ACC, blame Coach K.