Yes, you can just uninstall it and there shouldn't be any harm done. The music app running is normal, its just sitting in memory so it can be accessed faster. Don't worry about things running in the background. Most of them only take up memory and nothing else. If you want an easy way to see what might actually be truly running, hit the add button on the homescreen and hit shortcuts. Scroll down to Settings and then add Running Services. You can also grab the Battery Usage shortcut there.
Running Services will let you see what tasks are running in the background. Most of those need to be running, but if you feel that your battery is running unusually low, you can stop services through this interface. Only stop services from apps you download though, never any built in apps.
Battery Usage will let you see what things are running down your battery. If you want to be battery conscious, go here to see if any apps are using unusual amounts of battery. If they are, there is a way to force stop them by tapping on them.
To exit apps, just hit the back button until you get back to the homescreen. If the app doesn't let you do that, then there is an exit button, usually in the apps menu function.
