News Group Posts

slow system out of memory swapping

From: Bela Lubkin <belal@sco.com>
Subject: Re: What is causing my server to page?
Date: Fri, 10 May 2002 01:08:22 GMT
References: <2b30e5d8.0205020620.dbe94fb@posting.google.com>
<3cd1cb97$0$79562$8eec23a@newsreader.tycho.net>
<2b30e5d8.0205090139.741b15e6@posting.google.com> 

John Seed wrote:

> The next time I was able to catch it happening;
> 
> SCO_SV kettsco 3.2v5.0.5 i80386    05/09/2002
>  
> 09:37:59   freemem   freeswp availrmem availsmem (-r)
> 09:38:12        44    210256    124009       868
> 09:38:22        35    207000    124011      1057
> 09:38:30        40    203752    124007       988
> 09:38:50        49    201136    124011      1057
> 09:38:58        36    195616    124009      1038
>  
> Average         40    203552    124009      1002
> # ps -el|sort -rn +9
>  20 B      0 24450  3763  0  51 20 fb1236e0 1459692         -       ? 00:00:43 vfsd

> As you have probably gathered, I'm no expert at this, but I think the
> ps was telling me that process 24450 was using about 1.5gig of memory.
> 245450 is a Visionfs process;
> # ps -ef | grep 24450
>     root 24450  3763  1 09:01:27       ?    00:00:44 vfsd --profile /usr/vision/ vfsprofile
> 
> There is no good reason for Visionfs to be using this much memory. 
> 500meg of memory would be overspecced for the entire system on this
> server, let alone 2gig!  Putting more memory in won't fix anything,
> I'm sure this fault will spiral up and chomp up whatever memory is
> there.
> 
> The system sped up again after a few minutes.  I checked the VisionFS
> log;
> 2002/05/09 10:04:13.380 (pid 24450)
> SCO VisionFS(3.0) FATAL ERROR:
> The program has encountered an error that means it cannot continue.
> It will now exit. A technical description is given below to help
> establish the cause.
>  server/process/abort
> [unknown time]  process/error   SCO VisionFS(3.0)
> Process se3660 has had a fatal signal: SEGV - segmentation violation
>  
> Aborting process
> 
> This explains why everything works happily again.  The 24450 process
> turned into a <defunct> process.  I have a good dozen of these on the
> server, which I presume are all the aftermath of the same problem.
> 
> 
> My VisionFS level is 9.00.925 which I know is old, but this is quite
> an old legacy system which is pretty much frozen until it is replaced
> and I fear change when it comes to VisionFS.  Also, I don't get this
> on any other VisionFS installation.
> 
> Any other ideas as to what I can do to investigate/resolve this issue?

As John DuBois suggested, you can bandage this with a strategically
placed "ulimit" command.

It sounds like the vfsd process is spinning out of control in some sort
of memory allocation loop.  If it is like typical programs that fail
this way, it will spin out its entire loop, consume all of memory, then
die with a coredump.  Unfortunately, a process which is in the throes of
dumping core continues to hold all of its memory until the coredump
completes.  So the entire event looks like this:

  - program gets caught in an allocation loop

  - system-wide memory is depleted, eventually getting close to zero
    (you can see that availsmem got down to <1000 on your system)

  - at the low-water mark for memory, the spinning processes gets an
    allocation failure and dumps core

  - dumping such a large core takes a long time; especially since large
    portions of the dumping process have probably gotten shuffled out to
    swap, so this is a disk-to-disk copy, possibly on the same disk

  - during this long core dump, systemwide memory is very low, so other
    processes may fail (fortunately this can be somewhat
    self-correcting, since other processes that die leave a little more
    headroom)

  - eventually the spinning process finishes dumping core and system
    memory availability leaps upwards

In `sar -r` output an event will look like this: availsmem is cruising
along at a fairly stable high value, then it starts dropping rapidly.
The rapid drop continues until it gets down to a low water value
somewhere in the 0-1000 range.  Then it will stay near that low water
value for a long time (could be several minutes and might creep up
slightly during this time).  Finally, it will pop back up to the
original stable value, or possibly a bit higher.  Grotty ASCII graphic:

  |                                _____________      ^
  | ______ process                | done              |
  |       \ spins                 |                   |
  |        \ out                  |                   |
  |         \ of                  |                   |
  |          \ control            |               availsmem
  |           \ .                 |                   |
  |            \ .                |                   |
  |             \ .dumps core...__|                   |
  |              \____---___----                      |
  +-----Time------------------------------------      v

Please see the following article for a set of tools which allow size
limits to be established for every process on your system:

  http://groups.google.com/groups?selm=9804010746.aa13926@vagabond.armory

>Bela<
 

Comments?

Jump to Comments

Many of the products and books I review are things I purchased for my own use. Some were given to me specifically for the purpose of reviewing them. I resell or can earn commissions from the sale of some of these items. Links within these pages may be affiliate links that pay me for referring you to them. That's mostly insignificant amounts of money; whenever it is not I have made my relationship plain. I also may own stock in companies mentioned here. If you have any question, please do feel free to contact me.

Specific links that take you to pages that allow you to purchase the item I reviewed are very likely to pay me a commission. Many of the books I review were given to me by the publishers specifically for the purpose of writing a review. These gifts and referral fees do not affect my opinions; I often give bad reviews anyway.

We use Google third-party advertising companies to serve ads when you visit our website. These companies may use information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide advertisements about goods and services of interest to you. If you would like more information about this practice and to know your choices about not having this information used by these companies, click here.