June 2021 eLog

2021-06-28

Author: Colin

• Tested remotely:
  • self.aim = AimTTiEL302P ('ASRL/dev/psu1::INSTR')
  • It works now? Don't really understand why this wasn't working last Thursday, but w/e

• Encountered same problem as Colin on 2021-06-14, with double PSU (/dev/ttyACM1) this time
  • confusing, because we added the following line to /etc/udev/rules.d/serial-ports.rules on Thursday, which should have resolved this?
    • KERNEL=="ttyACM0", MODE="0666"
  • I tried restarting again, and then it worked... maybe multiple restarts is for some reason necessary to actually update serial-ports.rules? weird
• I'm wondering if maybe it would be best to assign the mode based on symlink as opposed to kernel, in the case that the PSU's are connected to different kernels (i.e. different usb ports), like so:
  • ATTRS{idProduct}=="049c", MODE="0666", SYMLINK+="psu1"
  • ATTRS{idProduct}=="0460", MODE="0666", SYMLINK+="psu2"
  • but for now this doesn't seem necessary

2021-06-24

Author: Colin, Laurel

• We are assigning a static port for each of the PSUs
• Using python to figure out the ports that each device is connected to
  • python
  • > import visa
  • > rm = visa.ResourceManager()
  • > rm.list_resources()
• The small PSU is currently on ASRL/dev/ttyACM0::INSTR
  • Which is apparently shortened to: /dev/ttyACM0
• Following directions at this link: https://msadowski.github.io/linux-static-port/
• We ran
  • SINGLE PSU: udevadm info --name /dev/ttyACM0 --attribute-walk
    • ATTRS{idProduct}=049c (under the second parent device)
  • DOUBLE PSU: udevadm info --name /dev/ttyACM1 --attribute-walk
    • ATTRS{idProduct}=0460 (under the second parent device)
• Then, like with solution from 6/14/2021 eLog, we added to /etc/udev/rules.d/serial-ports.rules
  • ATTRS{idProduct}=="049c", SYMLINK+="psu1"
  • ATTRS{idProduct}=="0460", SYMLINK+="psu2"
• After a restart, these showed up when we ran "ls /dev/psu*"
• However, there's something about using the specific ASRL*::INSTR names from the VISA manager library thing that means this isn't the full solution.. So need to think about this some more.

2021-06-22

Author: Colin

• When logging into odo remotely, getting these warnings about nut ad infinitum

  • Broadcast message from nut@ctalab-Precision-3630-Tower (somewhere) (Tue Jun 22
    
    Communications with UPS nevis-ctalab-ups@localhost lost
    
    
    Broadcast message from nut@ctalab-Precision-3630-Tower (somewhere) (Tue Jun 22
    
    Communications with UPS nevis-ctalab-ups@localhost established

  • When I run 'systemctl status nut-monitor.service', I get back:

    • Jun 22 16:57:18 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
      Jun 22 16:57:23 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established
      Jun 22 16:58:03 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
      Jun 22 16:58:03 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost lost
      Jun 22 16:58:08 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
      Jun 22 16:58:13 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
      Jun 22 16:58:18 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established
      Jun 22 16:59:08 ctalab-Precision-3630-Tower upsmon[14298]: Poll UPS [nevis-ctalab-ups@localhost] failed - Data stale
      Jun 22 16:59:08 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost lost
      Jun 22 16:59:13 ctalab-Precision-3630-Tower upsmon[14298]: Communications with UPS nevis-ctalab-ups@localhost established

    • Unclear what any of this means
  • Maybe this helps:
    • (Fix, but seems to assume a serial connection) https://www.truenas.com/community/threads/ups-repeatedly-gaining-and-losing-connection-with-server.40532/
    • There's also some documentation of the issue here, potentially as it relates to upssched:
      • https://github.com/networkupstools/nut/issues/219
      • Unclear if there's actually a resolution here as much as an acknowledgment of the issue.
    • Similar ideas here to make changes to timing things in conf files
      • https://www.truenas.com/community/threads/data-for-ups-is-stale.20898/
      • https://forum.openmediavault.org/index.php?thread/4131-solved-ups-communication-instable/
  • Advice from Bill
    • To suppress these messages, you want to edit the file:

      /etc/nut/upsmon.conf

      Do a search on NOTIFYFLAG. You probably want to remove the WALL option from something like:

      NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC

      Then restart upsmon:

      systemctl restart nut-monitor 

    • Nothing was set by default for NOTIFYFLAG in upsmon.conf
      • However, it says that the default behavior is that
        • "upsmon sends walls (global messages to all logged in users) and writes to the syslog when things happen. You can change this."
      • So I modified the file to say:
        • NOTIFYFLAG COMMBAD SYSLOG
      • The syslog appears to be located at /var/log/syslog
    • I restarted nut-monitor
    • Now am keeping an eye on /var/log/syslog with tail -f
      • It hasn't reported any stale data or lost connections yet, so I'm not thinking I should have seen anything in the "WALL" yet, but at the very least no notifications have shown up at the wall for the time being.
      • Looks like this was part of the way successful, got rid of the connection lost notification, but not the connection established ones
      • Modified /etc/nut/upsmon.conf now to include the line:
        • NOTIFYFLAG COMMOK SYSLOG
      • this looks like it fixed it entirely

2021-06-14

Author: Colin

• Encountering once again the same issues from the 2021-01-20 eLog.

  • Traceback (most recent call last):
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialposix.py", line 265, in open
        self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
    PermissionError: [Errno 13] Permission denied: '/dev/ttyACM0'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/dcps/SCPI.py", line 69, in open
        self._inst = self._rm.open_resource(self._resource,
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/highlevel.py", line 1771, in open_resource
        res.open(access_mode, open_timeout)
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/resources/resource.py", line 218, in open
        self.session, status = self._resource_manager.open_bare_resource(self._resource_name, access_mode, open_timeout)
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa/highlevel.py", line 1725, in open_bare_resource
        return self.visalib.open(self.session, resource_name, access_mode, open_timeout)
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/highlevel.py", line 194, in open
        sess = cls(session, resource_name, parsed, open_timeout)
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/sessions.py", line 218, in __init__
        self.after_parsing()
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/pyvisa-py/serial.py", line 65, in after_parsing
        self.interface = cls(port=self.parsed.board, timeout=self.timeout, write_timeout=self.timeout)
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialutil.py", line 240, in __init__
        self.open()
      File "/home/ctalab/anaconda3/envs/cta/lib/python3.8/site-packages/serial/serialposix.py", line 268, in open
        raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
    serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'

  • Going to try and use the second option from this answer on stack overflow: https://stackoverflow.com/a/27886201
    • Added it in at: /etc/udev/rules.d/serial-ports.rules
    • This seems to have resolved the problem
• Starting to build up the base of the gui at /home/ctalab/software/control_guis/psu_gui.py

2021-06-10

Author: Massimo

• Performed a visual inspection with Nancy's microscope of all the components on the board, no apparent signs of broken pieces (cracks, burns...)
• Talked to Leonardo, neither he nor Francesco L. have any better idea of what could have possibly failed; they'll try to find a spare board, re-program the FPGA and send it to us

2021-06-09

Author: Massimo with support from Ray

• We replaced the ADP3338 again, but there still is a short between Vout (which should be 3.3V) and ground. We then removed the ADP3338 and checked (short still there) the first immediate components downstream of the 3.3V
  • FC06
  • U3
  • FC01
• Nothing helped
• Asked Francesco L. for extra ideas and otherwise for spare board

2021-06-08

Author: Massimo

• Replaced U6 with new ADP3338; the power supply immediately reaches compliance, lowering voltage from input 5V to 2V roughly. Also, ADP338 immediately starts heating up, there is still a short somewhere. Check capacitors FC05/6?

-- Massimo Capasso - 2021-06-08

Comments