June 2021 eLog

2021-06-22

Author: Colin

  • When logging into odo remotely, getting these warnings about nut ad infinitum 
    • Broadcast message from nut@ctalab-Precision-3630-Tower (somewhere) (Tue Jun 22

Communications with UPS nevis-ctalab-ups@localhost lost


Broadcast message from nut@ctalab-Precision-3630-Tower (somewhere) (Tue Jun 22

Communications with UPS nevis-ctalab-ups@localhost established
    • When I run 'systemctl status nut-monitor.service', I get back: 
      • Jun 22 16:57:18 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
Jun 22 16:57:23 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established
Jun 22 16:58:03 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
Jun 22 16:58:03 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost lost
Jun 22 16:58:08 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
Jun 22 16:58:13 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
Jun 22 16:58:18 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established
Jun 22 16:59:08 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
Jun 22 16:59:08 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost lost
Jun 22 16:59:13 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established
      • Unclear what any of this means
    • Maybe this helps:
      • (Fix, but seems to assume a serial connection) https://www.truenas.com/community/threads/ups-repeatedly-gaining-and-losing-connection-with-server.40532/
      • There's also some documentation of the issue here, potentially as it relates to upssched:
        • https://github.com/networkupstools/nut/issues/219
        • Unclear if there's actually a resolution here as much as an acknowledgment of the issue.
      • Similar ideas here to make changes to timing things in conf files
        • https://www.truenas.com/community/threads/data-for-ups-is-stale.20898/
        • https://forum.openmediavault.org/index.php?thread/4131-solved-ups-communication-instable/
    • Advice from Bill
      • To suppress these messages, you want to edit the file: 
        /etc/nut/upsmon.conf
        Do a search on NOTIFYFLAG. You probably want to remove the WALL option from something like: 
        NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC
        Then restart upsmon: 
        systemctl restart nut-monitor
      • Nothing was set by default for NOTIFYFLAG in upsmon.conf
        • However, it says that the default behavior is that
          • "upsmon sends walls (global messages to all logged in users) and writes to the syslog when things happen. You can change this."
        • So I modified the file to say:
          • NOTIFYFLAG COMMBAD SYSLOG
        • The syslog appears to be located at /var/log/syslog
      • I restarted nut-monitor
      • Now am keeping an eye on /var/log/syslog with tail -f
        • It hasn't reported any stale data or lost connections yet, so I'm not thinking I should have seen anything in the "WALL" yet, but at the very least no notifications have shown up at the wall for the time being.
        • Looks like this was part of the way successful, got rid of the connection lost notification, but not the connection established ones
        • Modified /etc/nut/upsmon.conf now to include the line:
          • NOTIFYFLAG COMMOK SYSLOG
        • this looks like it fixed it entirely

2021-06-14

Author: Colin

  • Encountering once again the same issues from the 2021-01-20 eLog. 
    • Traceback (most recent call last):
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialposix.py", line 265, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
PermissionError: [Errno 13] Permission denied: '/dev/ttyACM0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/dcps/SCPI.py", line 69, in open
    self._inst = self._rm.open_resource(self._resource,
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/highlevel.py", line 1771, in open_resource
    res.open(access_mode, open_timeout)
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/resources/resource.py", line 218, in open
    self.session, status = self._resource_manager.open_bare_resource(self._resource_name, access_mode, open_timeout)
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/highlevel.py", line 1725, in open_bare_resource
    return self.visalib.open(self.session, resource_name, access_mode, open_timeout)
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/highlevel.py", line 194, in open
    sess = cls(session, resource_name, parsed, open_timeout)
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/sessions.py", line 218, in __init__
    self.after_parsing()
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/serial.py", line 65, in after_parsing
    self.interface = cls(port=self.parsed.board, timeout=self.timeout, write_timeout=self.timeout)
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialutil.py", line 240, in __init__
    self.open()
  File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialposix.py", line 268, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'
    • Going to try and use the second option from this answer on stack overflow: https://stackoverflow.com/a/27886201
      • Added it in at: /etc/udev/rules.d/serial-ports.rules
      • This seems to have resolved the problem
  • Starting to build up the base of the gui at /home/ctalab/software/control_guis/psu_gui.py

2021-06-10

Author: Massimo

  • Performed a visual inspection with Nancy's microscope of all the components on the board, no apparent signs of broken pieces (cracks, burns...)
  • Talked to Leonardo, neither he nor Francesco L. have any better idea of what could have possibly failed; they'll try to find a spare board, re-program the FPGA and send it to us

2021-06-09

Author: Massimo with support from Ray

  • We replaced the ADP3338 again, but there still is a short between Vout (which should be 3.3V) and ground. We then removed the ADP3338 and checked (short still there) the first immediate components downstream of the 3.3V
    • FC06
    • U3
    • FC01
  • Nothing helped
  • Asked Francesco L. for extra ideas and otherwise for spare board

2021-06-08

Author: Massimo

  • Replaced U6 with new ADP3338; the power supply immediately reaches compliance, lowering voltage from input 5V to 2V roughly. Also, ADP338 immediately starts heating up, there is still a short somewhere. Check capacitors FC05/6?

-- Massimo Capasso - 2021-06-08

Comments

This topic: Veritas > PSCT > NevisLabsSetup > NevisLabsELog > NevisLabsELogJune2021
Topic revision: r8 - 2021-06-23 - ColinAdams
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback