To get specific, it is true that the Church is "A body of men united together by the profession of the same Christian Faith, and by participation in the same sacraments, under the governance of lawful pastors, more especially of the Roman Pontiff, the sole vicar of Christ on earth." (See here for a fuller definition.)
Another definition of the Catholic Church is found here by Fr. John A. Hardon, S.J. He says, "By comparison with Protestant definitions of the Church, Bellarmine explains that ours is radically different for one basic reason. 'All the rest require only internal virtues to constitute a person in the Church, and, therefore, they make the true Church something invisible. Whereas we also believe that in the Church are found all the virtues: faith, hope and charity, and the rest,' However, for anyone to be called in some sense a part of the true Church, of which the Scriptures speak, what is essential is 'an external profession of faith and communication of the sacraments, which can be perceived by the senses themselves. For the Church is an assembly of men, as visible and palpable as the assembly of the Roman people, or the Kingdom of France, or the Republic of the Venetians.'"
|Ivory tabernacle (the Louvre)|
Additionally, the roots of the Catholic Church derive from the Jewish faith. When the Jews were wandering in the desert at the time of Moses, they carried a moveable tabernacle that was organized by God in very strict terms. This was the physical dwelling place of God with His people, and was called the Tent of Meeting. In the time of David and Solomon, a permanent structure--the temple was built, again according to detailed instructions from God. The red sanctuary lamp in Catholic churches, to give just one example, has its roots in the Jewish tabernacle.
Our Catholic churches are a visible presence in our communities. Their very architecture testifies visibly to Christ's presence with us. According to Fisheaters, "Catholic vision assigns symbolic meaning to the various parts of the church building, as it does to pretty much everything else in the world. The roof symbolizes charity, which covers a multitude of sins; the floor symbolizes the foundation of faith and the humility of the poor; the columns represent the Apostles, Bishops, and Doctors; the vaulting represents the preachers who bear up the dead weight of man's infirmity heavenwards; and the beams represent the champions of ecclesiastical right who defend it with the sword. The nave symbolizes Noah's Ark and the Barque of St. Peter, outside of which no one is saved. The direction of the East represents the Heavenly Jerusalem, and the direction whence the Messiah will return in glory; West represents death and evil."
According to the How-To Book of the Mass, "There is great symbolism in the doors of a church...Your church may or may not have ornate doors, but the symbolism of what the door signifies remains. Sometimes you will see the image of Christ holding a lamb on his shoulders painted or depicted in stained glass over the front of the doors of a church. This reminds us that Jesus said, 'I am the door; if anyone enters by me, he will be saved, and will go in and out and find pasture.'" (John10:9).
According to Deacon Dean, the iconography of our churches, "...is a huge part of our Catholic tradition. It communicates our heritage and faith." See here, here, and here for his series of ongoing articles about the iconography of St. Mary's in the All Saints Parish church bulletin.
A Catholic church is simply not like the facilities of other Christians. Inside and out, top to bottom, and most importantly because in them Christ Jesus is present in the Eucharist, Catholic churches are set apart. Unless special circumstances warrant it (military situations, a destroyed church, etc.) according to canon law (932), mass is to be said in a Catholic church. That is the norm and expectation in the vast majority of cases. In fact, if a Catholic church has been desecrated, worship cannot go on there again until a penitential rite has been said (Canon 1211-1212).
The mass, as described by Lumen Gentium fromVatican II is "the source and summit of the Christian life." And according to the Catechism of the Catholic Church, "The other sacraments, and indeed all ecclesiastical ministries and works of the apostolate, are bound up with the Eucharist and are oriented toward it. For in the blessed Eucharist is contained the whole spiritual good of the Church, namely Christ himself, our Pasch." (CCC 1324)
In short, you cannot go just anywhere to worship and receive the Eucharistic blessing that is bestowed on Catholics during worship at mass.