![[lang]](/images/eye.png)
CentOS debug fail |
2009-04-30
|
Recently we got a new server for apestaart, our little hosting project between friends.
This time we decided to install CentOS, since people I trust have been saying good things about it, we use RHEL at work and it's pretty much the same, and Wiebe, the other admin, also was all for it.
This week I ran into a segfault and I wanted to debug it. Turns out CentOS ships without any debug repositories installed by default. Fedora installs the files, but cleverly disables them, as they should. But yumutils contains debuginfo-install, a handy script that allows you to install all the dependencies of a -debuginfo package, and gdb spits out useful commands like what packages to install when you're looking at a stack trace with missing symbols.
So, this doesn't work out of the box on CentOS. Fail #1.
I went on IRC, mentioned this there, and I was told that 'most CentOS users don't need debuginfo packages'. Fail #2 - neither do Fedora users, yet somehow Fedora managed to figure out how to work both for the ones that don't need them and the ones that do. So goodbye unhelpful IRC channel.
I had to manually create a .repo file based on some guy's post complaining about the same thing; something much like this:
[root@betsy ~]# cat /etc/yum.repos.d/debuggery.repo
[debuggery]
name=CentOS-$releasever - DebugInfo
baseurl=http://debuginfo.centos.org/$releasever/$basearch/
priority=1
Then I actually tried to do debuginfo-install python, and what happened ?
First of all, the python-debuginfo package for the installed version of python isn't even in that repo (though 3 others are, go figure). FAIL #3
Of the other packages that it did find, one was unsigned:
yum.Errors.YumBaseError: Package libtermcap-debuginfo-2.0.8-46.1.x86_64.rpm is not signed
FAIL #4.
CentOS, hostile to developers. Back to old-school 90's style rpm hunting and pecking on the web.
UPDATE:
As for the python debuginfo package, there is a RHEL5 package with exactly the right version and release tag, but that one was built for RHEL5, and installing it mismatches against the CRC of the installed python package. CentOS simply does not provide the debuginfo package for its shipped version of python. FAIL #5
13:08 Hi. Two simple questions: a) why is there no default config for the debuginfo repo’s in the base install, so that debuginfo-install Just Works ? 2) why is Package libtermcap-debuginfo-2.0.8-46.1.x86_64.rpm is not signed
13:10 homeasvs__: 90% of normal users don’t need (or know what to do with) debug packages
Comment by you fail — 2009-04-30 @ 21:04
@you fail: not sure why you’re pasting the IRC logs that shows exactly what I said in the post ?
Comment by Thomas — 2009-04-30 @ 21:09
Compared to what you were told, this looks to be a misquoted quote.
… that ‘most CentOS users don’t need debuginfo packages’.
Comment by you fail — 2009-04-30 @ 22:32
Yes, the debuginfo packages are an issue with CentOS. They take a huge amount of storage, pushing it to mirrors consumes precious bandwidth and even more storage space. And given the fact that only very occasionally people request them, it does not have a big priority within the CentOS project. So it is no surprise the yum repo config is not available.
That doesn’t mean it cannot be improved and some of the issues you encountered should never have been the case in the first place. But it is a fact that if we had more resources, they would probably be used somewhere else and not on debuginfo packages. The return-on-investement (return in a volunteer project is not measured in a currency) makes it less of a priority. Even when I would have preferred this being a non-issue.
Maybe a post like this makes a difference, we can certainly hope for it.
Comment by Dag Wieers — 2009-04-30 @ 22:33
After some investigation this is what I can tell.
– The CentOS project depends on hardware donations from users and unlike Red Hat or Fedora has no budget nor an ideal infrastructure. The donated systems and bandwidth are available over different datacenters on this planet.
– Because of this it is quite difficult to sign debuginfo packages like normal packages as they have to be transferred (which is practically impossible due to the sheer size) and would slow down releasing normal updates. Some technical solutions have been discussed on centos-devel and will be implemented soon
– That is also the reason why these packages are only available from one system, and are not transferred to mirrors. Because of this and the fact that only occasionally someone needs them, it does not get a lot of priority.
Now, it was already known that debuginfo packages are lagging behind normal updates. After the CentOS 5.3 release this being resolved slowly. But apparently the centos-devel mailinglist did have some discussions in improving the debuginfo setup. Including adding a yum debuginfo config file once the other issues are fixed.
PS Next time we would appreciate a message to the mailinglist, or rather a bug-report instead. Much like any Open Source project our volunteers prefer positive criticism and feedback over this.
Comment by Dag Wieers — 2009-04-30 @ 23:25
@you fail: still don’t follow. How is ‘90% of normal users don’t need (or know what to do with) debug packages’ different from ‘most CentOS users don’t need debuginfo packages’ ? To me ‘90%’ and ‘most’ are pretty close.
@dag: I understand that it is a volunteer-run project, and that not everything can be expected. However, as such, it simply means that it is not a friendly system to developers, not as a user who wants to develop, and not as a developer who would like to get good bug reports from their users. FWIW, the segfault I’m running into only happens on CentOS 5.3, not on RHEL5.3 (with same versions of packages involved, although rebuilt), and not on Fedora. (https://bugzilla.redhat.com/show_bug.cgi?id=498512) The lack of working debuginfo packages makes it just that harder to debug.
Comment by Thomas — 2009-04-30 @ 23:37
Thomas, the only thing I can add is that we are working on those issues. But there were understandable reasons for some of the fails you encountered (no yum config, and not signed) which were being tackled. The fact that some packages were missing was already known and is a fairly recent problem (since the 5.3 release).
That said, I do not agree with the tone of the message you delivered. If this is how issues are reported in the Open Source community then I am pretty sure we wouldn’t be as successful as we are. Besides generalizing that CentOS is hostile to developers, while you had a particular problem is a bit stretched. But no feedback is worse than this, so nevertheless thanks for that ;-)
Comment by Dag Wieers — 2009-05-01 @ 11:51
It amazes me that people in the open source community can treat others with such disrespect. This is no different than new linux users demanding developers fix the bugs they care about, as if they were paid employees who owed something to the user.
There is nothing wrong with pointing out where CentOS is lacking, but you make it sound like they are somehow incompetent and you reward their volunteer work, volunteer work that makes the life of thousands of people easier, by mocking them viciously.
Perhaps you should try and put yourself in their perspective before treating volunteers so badly.
Comment by shawn — 2009-05-01 @ 16:01
@shawn: I’ll take your opinion in consideration. However, I do think it is incompetent to fork an existing OS and not do it correctly. I don’t think being a volunteer or not is the reason. First of all, on IRC they didn’t say “Yes, that’s a problem we have that we should fix; care to help ?”. They said “users don’t need them”, thereby implying that my need for a debuginfo package was unreasonable. Second, Debian seems to run just fine on volunteers. I also don’t agree with the implication that it would be OK to treat paid workers with disrespect.
Finally, my biggest problem with this is exactly the same problem you seem to have with my disrespect (which was not intended as disrespect): your opinion on open source. It is my opinion that open source is successful exactly because the distributions are transparent, easy to debug, and help to enable users to provide their developers with good bug reports. I’m surprised to see that people don’t agree that providing good debuginfo packages, so users can provide good backtraces, so open source hackers like me can fix bugs, is a part of that.
Remember – the only reason I ended up with this problem is exactly because I was trying to fix an actual bug in an actual open source application on an actual open source distribution.
Comment by Thomas — 2009-05-01 @ 17:02
Thomas, CentOS is not hostile to developers. I am disappointed at you.
You put it so mildly. They are hostile to EVERYONE. Nothing but a community project. They are bunch of snobs looking down on everyone. The whole site was controlled by a single person for years (you could see from the updates happenning once in 2 years) and there were some security breaches in their servers. Their response to a user inquiring about the extend of it made my jaw drop. It was basically “Get lost”.
I have a script that pulls down SRPMS from RedHat and does automatic rpmbuilds. I have a limited set of RPMS on my servers. This serves me very well. RedHat does not require removal of their installation after trial. If you are doing your own rpmbuilds, you are good to go. That is what they do anyway. It is not as if they are producing anything extraordinary.
I faced a similiar situation like you some time ago when I wanted to try CentOS. I was having segfaults on KS DHCP installs (a problem that I did not have on RHEL). After their responses, I decided that it is not a community I want to be associated with. They have no idea about “COMMUNITY” or “VOLUNTEER WORK”.
And you Dag… I am sure you got a taste of your poison after all the crap they told you before and after you left (Thank you for your excellent RPMS by the way).
Comment by Jimmy — 2009-12-05 @ 23:56
My log files indicate this was said by a nick containing ‘homeasvs’ as to debuginfo files
[herrold@nfs freenode]$ grep homeasvs \#centos.log | grep -v “\[”
14:08 homeasvs__> Hi. Two simple questions: a) why is there no default config for the debuginfo repo’s in the base install, so that debuginfo-install Just Works ? 2) why is Package libtermcap-debuginfo-2.0.8-46.1.x86_64.rpm is not signed
14:10 Evolution> homeasvs__: 90% of normal users don’t need (or know what to do with) debug packages
14:12 donavan> homeasvs__: what directory are the debuginfo packages stored in?
14:47 homeasvs__> Evolution, that’s a strange reason for not having them there by default in fedora
14:48 homeasvs__> donavan, http://debuginfo.centos.org/5/x86_64/
14:49 homeasvs__> doesn’t look like the debuginfo repo is in sync either – 4 python-debuginfo rpms there and all of them older than the actual python rpm available
I have no idea what Fedora has to do with the questions asked; times are US ET
Comment by herrold — 2010-06-15 @ 19:02
CentOS may not be hostile towards developers, but it is definitely catering more toward Systems/MIS people than developers. Most of the loosely defined CentOS core teams are UNIX admins, not developers, so they worry about the problems that are germane to UNIX admins. Dropping into gdb to debug code is not something they know how to do, so they just assume that nobody would ever want to do it.
Having said that, some of the admins on the CentOS IRC channel are hostile and short-tempered, not just to developers, but just about everyone, so don’t take it personally.
Comment by Matt Rose — 2010-06-18 @ 21:04