Are some languages more or less difficult to learn than other languages?

The other day, I went over Pats house. We eventually ended up at his grandparents house. They only spoke Polish. I was trying to listen to their conversation, but I couldn't pick up anything they were saying. It made me think about whether it was easier to learn some languages than others. I tend to believe that depending on when they are learned, no language is harder than another. If someone is learning Polish as their first language, it will be no harder for them than learning English or Spanish first. However, if it is not learned right away, then there might be a difference. It will probably be easier for me to learn a language closer to Englishthan something very different, like Polish. It all depends on the tije that the language is learned.