Figuratively, to open a can of worms is to create or initiate a situation that will cause trouble or will simply be unpleasant.
Worms are considered unpleasant and rather dirty in English culture. To open a can of worms - such as worms prepared as bait for fish - makes cleaning things up again much more difficult. Just as letting the cat out of the bag means letting a secret out that cannot be hidden again, opening a can of worms means starting trouble that is much more difficult to finish, than it was to start.
Example: "When Jim said bad things about Megan's web page, little did he know that he was opening a can of worms. The fans of Megan's page were convinced that the criticism was unfair and mean-spirited. Jim ended up receiving an avalanche of criticism himself." In this example, the initiator of trouble gets far more trouble than he bargained for (than he believed he was going to receive).