Dan's I.T. notes

• definition (upgrades, installations, corrections, etc)
• use RCS and/or CVS
• production change logs for each host
• regular meetings, even if just for updating of "status dashboard".

The objective should be to minimize interruptions of our poor dear BOFH. Those user requests which are minor or redundant can be cached here. Nontrivial issues can be tracked. Understanding of repetitive problems can be researched. BOFH-in-training has an easier time.

• trouble ticket system such as Double Choco Latte (http://dcl.sourceforge.net)
• user-servicable p.r. interface, for submission and followup queries
• manager-based delegation and prioritization per item
• a primary interface between sysadmin and users

Communication should be strengthened in whatever means, via work or play. Meetings and recreation breaks should be somewhat mandated by the manager, include most of the staff most of the time, and should be in a low-stress environment (amusement park, restaurant, golf course, luxury cruise, massage house, etc). Especially in a start-up, this kind of thinking should be standard company issue to all employees. They should have no trouble deciding whether or not to get up each morning.

• pseudo-regular meetings
• mailing lists, IRC, prms
• stress relief
  • rental of DVD or video game system, installed in rec room on large display and excellent sound
  • sports
  • get a projector if you can afford it
  • on-site catering, massage, chiropractic, and other routine personal needs
  • Quake, DOOM, etc with custom levels based on work environment

• Change root passwords
• personal maintenance [ongoing]
  • Consolidate and organize all personal bookmarks.html and publish if necessary.
  • Make a checklist of your informational devices.
    • email
    • voicemail
    • job ticket system
    • your home directory hierarchy (~/doc)
    • intranet departmental docs
    • bookmarks.html on your various systems (home, office, etc)
  • Periodically consolidate and organize all incoming information.

    Have a storage model that can be consistant across each storage method. In other words, you can have the same categories for your email folders, directory structure in your home directory, the job ticket system, etc. The items stored in each can be appropriate for that storage method. For example, when I order something online, I save the emailed purchase order in my "order_status" email folder. Then I check my order status online and I might print that to a Postscript file in ~/doc/order_status. When I receive the item, I would put the printed invoice in a folder.

    Make enough categories in every storage method that you can immediately file every incoming thing according to its most objectively relevant primary category. Create, rename, and delete them on the fly if you like. Here are example folders for your email, home dir, and job ticket system.

    • it_personnel
    • engineering
    • it_info
    • it_request
    • gnu_info
    • it_dcl
    • it_inventory
    • it_shopping
    • order_status
    • telecom

• DNS Client if not DHCP Client
  • search path of internal domain, then external domain
• Authentication and User Info Client
  • See Authentication and User Info Server

([client mod] indicates that modifications will be needed on the client host(s))

• Security Server
  • firewall
    • upgrade firewall kernel[ongoing]
    • intrusion and detection system (IDS)
    • ipchains and kernel 2.2 until 2.4 is stable
    • consider firestarter.sf.net -- GUI config wizard, forward compatible with iptables
    • no outgoing spoofing -- outgoing source address restricted to your own domain
    • add logging of privileged and suspicious ports
    • public service from private servers
    • snmp client, logged to a web page on the intranet server
    • preferably with all readonly filesystems, preferably bootable from CD
  • VPN
  • sshd
    • only for remote administration
    • no remote root logins
• Proxy Server
  • Squid proxy on tcp 8080
  • optional ad removal (Junkbuster and Dan's Guardian) on another port
  • Can force the use of a proxy using transparent proxying in the Linux kernel of the firewall
• Monitoring Server
  • find a secure remote syslog implementation
  • copy all nontrivial syslog from all hosts directly to this host
  • netsaint
  • modem and analog phone line directly attached, with which to perform paging in case of emergency -- or perhaps use the emergency modem on the Security Server
  • if an IT service provider is involved in the monitoring, and if the logs files are under a certain size, periodic RSA-authenticated scp of various logs to their site
  • Cacti for monitoring and graphing
  • Nessus for triggered via email or via a dialout modem
  • automatic analysis of security logs
  • monthly proactive security breakin attempts
  • upgrade of security vulnerabilities as they're discovered (ongoing)
  • some or all of the web interface to these apps should be hosted on or linked from the Intranet Server
• Backup Server
  • try to use backup software which reports via snmp
  • see File Server and Authentication notes on storing everything on the server for centralized backups
  • try to have a physically separate backup server due to reliability of the device, device drivers, and . You don't
  • reliable removable media -- either tape or cd-r, and no hard drives
  • rotation and offsite storage policy
    • at least monthly full and daily differential
    • contact insurance company about their policy on offsite archival
  • investigate Arkeia (http://arkeia.com)
  • small clients with no onsite backup can do secure remote backup to IT providers' site
• DNS Server [client mod]
  • DNS entry for each client, hostname.internaldomain
• Virus Scan Server
  • actively scan all Internet traffic at the gateway via ftp, http, and smtp proxy
  • clamav
• DHCP Server [client mod]
• • propagate DNS, default gateway, hostname, ntp server (time.apple.com)
• Mail Server
  • IMAP
  • POP3
  • spam blocking
  • virus scanning (see Virus Scan Server
  • web interface; see Intranet Server
  • email autoresponder for role accounts like 'sales' or 'jobs'
  • users authenticate to IMAP and their IP addresses are automatically added to smtp relay rules; built into postfix
• Authentication and User Info Server [client mod]
  • LDAP server as the root of all info
    • PAM on each host authenticating to LDAP
    • most apps that aren't PAM enabled can authenticate directly to LDAP
    • any remaining apps can probably use a relatively secure script running on the LDAP server to export passwords in each app's format
  • What's the feasibility of using a secure physical object for authentication? keycards, etc
• File Server
  • usage
    • exported by NFS, Appletalk, and SMB
    • home directories
    • CVS data
    • MP3
      • all the cool radio-station-like stuff from freshmeat.net, slashdot.org, etc
      • convert current file format collections (.au, .wav, etc)
      • http://www.icecast.org
  • implementation
    • If this is only a file server, check out the Samba docs for memory optimization toward filesystem cache
    • ACL support
    • ext3 or xfs
    • netatalk [client mod]
    • Samba[client mod]
      • buy a book
      • proper file locking for stupid Windows programs such as Quickbooks and ACT
      • optional encrypted password support via cron passwd conversion
    • knfsd[client mod]
    • • latest server kernel, hopefully on client as well
      • server for desktop home dirs
    • make CNAME for file server: home
    • user enviroment
      • umask 002
      • edit users' dotfiles to source a system-wide environment file first
      • CVSROOT and other environment variables
      • ulimit 0
    • fix Windows lockfile ownership via any of the following:
      • chattr
      • Samba
      • brute force cron chmod
      • parent directory modes
    • filesystem groups per user function
      • finance
        • move Quickbooks here
        • verify proper accounting/i.t. protocol on backups, location, and file sharing between users of virtually inevitable crappy proprietary software
      • www
• web server
  • web site log analysis and retention
• print server [client mod]
  • proper banner page
  • try to account or predict toner and paper usage
  • add either VMware with Windows as the print server
  • or the HP Jetadmin suite running on a Sparc with either Solaris or Linux and Solaris libraries
• ntpd [client mod]
  • install ntpd client on all servers
  • may require individual host 'date' configurations in order to adjust potentially major clock skew in order for ntpd to function

• mail [client mod]
  • Linux client /etc/sendmail.cf mods for rewriting outgoing headers to username@yourdomain
• Virus Scanning
  • clamav and http://www.antivirus.com/free_tools/
• Migrate MacOS and Windows users to fresh setup
  • if their current filesystem is really wacky, install all their current apps on vnc vmware, have them test it for a while.
• Linux authentication
  • local accounts in /etc/passwd
    • 'root' for local emergency only
    • 'it' for automatic remote administration
    • RSA authenticated ssh key
  • mail mta
    • Linux client /etc/sendmail.cf mods for rewriting outgoing headers to username@yourdomain
  • LDAP client
    • all employee user accounts
• Windows installation
  • bare custom mod Windows install
  • add apps
  • add Samba config
    • automount home export and any group exports
    • add "Apple Laserwriter" postscript printer support
    • add occasional local hardware drivers
    • 'tar', NFS, and Slackware root/boot for Windows installations
  • security
    • freely distributable ssh clients
      • .deb, .rpm, .tgz, and binary dists for Linux
      • Nifty Telnet for MacOS
      • find one for Windows/DOS, such as on ftp.ssh.com or on the VNC homepage
  • misc
    • network autoconfig (DHCP, bootp, tftp, NIS user auth)
    • 'tar' and misc commands/files to install others
    • common software repositories and package management (see 'company-wide' section)
    • Check any company-wide audits such as y2k, to see if we need to consolidate users' needs to prevent them from having to buy lots of I.T.-unsupported things on their own.
    • external shell server in DMZ for testing of various public conditions, possibly for employee use

• Listing on PacBell directory service.
• toll free numbers
• get a local PBX or an account on local telco's PBX, for handling of voicemail, etc

• Groupware
  • Applications
    • See Exchange Obsolescence Initiative
      • shared calendaring, mail folders, and contacts via open standards: ftp, imap, etc
    • See Outlook Obsolescence Initiative
      • See Exchange Obsolescence Initiative
    • imap webmail
      • phpGroupWare's "SquirrelMail"
    • shared calendar / scheduling
      • phpGroupWare's "calendar"
    • contact management
      • phpGroupWare's "addressbook"
      • replace something like ACT!2000
      • bidirectional synching between PIM app and device
        • web app
        • palm pilot
        • mobile phone
      • import from other contact apps like ACT!2000
      • contact history
      • statistical analysis or visualization (graphs)
    • simple "to do" list
      • phpGroupWare's "to do"
    • news and bulletin board
      • phpGroupWare's "wcm"
    • project management, work flow, and job ticket
      • phpGroupWare's "dclgw" or Double Choco Latte
      • replace something like Project(tm)
    • facilities reservation
      • phpGroupWare's "Resource Booking System", being merged into "calendar"
    • filesystem/document management
      • phpGroupWare's "file manager"
      • document format converter
        • use 'wv' as a cgi to convert .DOC files into html
      • mailing lists
        • list server, with web interface for user-level management of mailing list accounts and new (personal) lists, and archiving
        • GNU Mailman
        • employee-facilitated tutoring and tech support (archiving)
        • jokes, rumors, B.S.
        • departmental announcements/discussion
        • I.T. accounts such as 'hostmaster', 'postmaster', 'help', etc.
      • snmp network log analysis page, like mrtg (see the ucd-snmp site)
      • Squid proxy log analysis page
      • web-based ssh client such as Mindterm for Java or whatever's built into webmin
      • IRC
        • standard channel and bot on irc.openprojects.net (with donation to OPN), or internal ircd
  • Global Features
    • awareness of organizational units / groups / departments
    • awareness of whereabouts and locations
    • user entities
    • open standard SSL encryption of at least all authentication data
  • Externally Available Features
    • login off of public web server
    • Remote Access to Internal Desktops
      • sysadmin assigns one tcp port on the Security Server, and one desktop hostname per internal desktop user, assigns LDAP fields for that info
      • internal desktop user logs into intranet SSL web page and clicks on "view your internal desktop" link, selecting whether they want to use their local VNC client or to download the Java client (find encrypted VNC clients for MacOS, Win32, Linux, and web browser JVM)
      • cgi on the intranet server queries LDAP for the user's tcp number and spawns an RSA-authenticated ssh command on the firewall that opens that user's tcp number for their remote IP address and redirects it to their internal desktop's VNC server, for 30 seconds
      • user (or maybe the cgi) spawns a local encrypted VNC viewer and uses their internal desktop host from anywhere

• documentation for procedures, policies, general computing, and new employees
• fun links
• local geography and travel links
• employee personal web sites, public and/or private bio's, etc

Inventory every nontrivial I.T.-related equipment per individual. Make labels with company name, phone number, and serial numbers. Print them onto a permanent adhesive pouch. The pouch contains the person's userid to whom the item is checked out.

• base desktop systems
• monitors
• base laptop systems
• nontrivial removable laptop components (cdroms, floppies, batteries)

• perhaps a deal with ISP for home VPN
• encrypted IMAP/POP3
  • optional web interface

• sudo
• RSA-authenticated ssh
• administrative filesystem groups

• encourage the porting of Stackguard to various architectures
• upgrade services
  • ssh
  • bind
  • Apache
• run security scanners such as SAINT and nmap
• firewall configs
• uninstall unnecessary services and samples
  • disable or secure public anonymous ftp
• disable unnecessary info (software version queries)
• whitelist configurations: ACLs, firewall, process and filesystem ownership

• kernel tuning

• installation and configuration
  • GNU configuration engine (http://www.iu.hioslo.no/cfengine/)
  • package management
    • RPM (http://www.rpm.org)
    • pkgadd (http://www.sunfreeware.com)

• try Arkeia

• UPS installation
  • get on-site UPS for at least the data center (web server)
• power termination (surge protection)
• electrical layout of the building
• power failure plan
  

  We may not want the whole building powering back on simultaneously (spike). Switching them off during the power loss can prevent the local building's power supply from failing. Also, some employees are necessary to have on-site and most are in the way at such a time. Fall back on roaming plan.

  • power-down procedures
    • noncritical services, desktops, printers, copiers, appliances
    • broadcast employee announcement via PBX
  • email to offsite employees (remote offices, home workers, etc)
  • non-infrastructural employees collect in one spot or go home (marketing, sales, etc)