OpenIndiana hipster 2014.1

July 10th, 2014

I haven’t been paying close attention the last few weeks and was surprised to discover that hipster stopped updating. So, I finally had time to go read the mailing list and discovered that they’d moved to a hipster-2014.1 repository that contains just the most recent packages.

Needed to set up my boxes at work to mirror this repository, so I created a new local mirror:

# mkdir /data/pkg/hipster-2014.1
# pkgrepo create /data/pkg/hipster-2014.1
# pkgrecv -s -d /data/pkg/hipster-2014.1 '*'
# svccfg -s pkg/server add hipster-2014-1
# svccfg -s pkg/server:hipster-2014-1 addpg pkg application
# svccfg -s pkg/server:hipster-2014-1 setprop pkg/port=10087
# svccfg -s pkg/server:hipster-2014-1 setprop pkg/inst_root=/data/pkg/hipster-2014.1
# svccfg -s pkg/server:hipster-2014-1 addpg general framework
# svccfg -s pkg/server:hipster-2014-1 addpropvalue general/complete astring: hipster-2014.1
# svccfg -s pkg/server:hipster-2014-1 addpropvalue general/enabled boolean: true
# svccfg -s pkg/server:hipster-2014-1 setprop pkg/readonly=true
# svccfg -s pkg/server:hipster-2014-1 setprop pkg/threads=100
# svcadm refresh application/pkg/server:hipster-2014-1
# svcadm enable application/pkg/server:hipster-2014-1

Then, I have to fix apache on the box by editing proxy.conf:

# vi /etc/apache2/2.2/conf.d/proxy.conf

I add the following line:

ProxyPass /hipster-2014.1 nocanon

Making the file look like:

ProxyRequests Off
ProxyVia Block
ProxyStatus On
ProxyPreserveHost Off
ProxyPass /dev nocanon
ProxyPass /sfe nocanon
ProxyPass /sfe-encumbered nocanon
ProxyPass /local nocanon
ProxyPass /legacy nocanon
ProxyPass /hipster nocanon
ProxyPass /hipster-2014.1 nocanon
AllowEncodedSlashes NoDecode

Restart apache, set the publisher, and update:

# svcadm restart svc:/network/http:apache22
# pkg set-publisher -p
pkg set-publisher:
  Updated publisher(s):
# pkg set-publisher -p
pkg set-publisher:
  Updated publisher(s):
# pkg image-update
           Packages to install:   3
            Packages to update: 487
           Mediators to change:   2
       Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                            490/490     7415/7415  674.6/674.6 13.5M/s

PHASE                                          ITEMS
Removing old actions                       2682/2682
Installing new actions                     3920/3920
Updating modified actions                10263/10263
Updating package state database                 Done
Updating package cache                       487/487
Updating image state                            Done
Creating fast lookup database                   Done
Reading search index                            Done
Building new search index                  1236/1236

A clone of openindiana-41 exists and has been updated and activated.
On the next boot the Boot Environment openindiana-42 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

NOTE: Please review release notes posted at:

And reboot, and then you have a new hipster box.

Building GNU Screen 4.2.1 on OpenIndiana

April 30th, 2014

Surprisingly, a new version of GNU screen (4.2.1) was released recently, so I thought I’d build it on my OpenIndiana (hipster) box to see if worked, and ran into a bit of a problem:

gcc -c -I. -I.  -DETCSCREENRC='"/usr/local/etc/screenrc"' -DSCREENENCODINGS='"/usr/local/share/screen/utf8encodings"' -DHAVE_CONFIG_H -DGIT_REV=\""`git describe --always 2>/dev/null`"\" \
     -g -O2 socket.c
socket.c: In function 'ReceiveMsg':
socket.c:990:16: warning: assignment from incompatible pointer type [enabled by default]
socket.c:994:6: error: 'struct msghdr' has no member named 'msg_controllen'
socket.c:995:6: error: 'struct msghdr' has no member named 'msg_control'
socket.c:1007:14: error: 'struct msghdr' has no member named 'msg_controllen'
socket.c:1010:14: warning: assignment makes pointer from integer without a cast [enabled by default]
socket.c:1010:48: warning: assignment makes pointer from integer without a cast [enabled by default]
socket.c: In function 'SendAttachMsg':
socket.c:1801:6: error: 'struct msghdr' has no member named 'msg_control'
socket.c:1802:6: error: 'struct msghdr' has no member named 'msg_controllen'
socket.c:1803:8: warning: assignment makes pointer from integer without a cast [enabled by default]
socket.c:1807:3: warning: passing argument 2 of 'bcopy' makes pointer from integer without a cast [enabled by default]
In file included from os.h:83:0,
                 from screen.h:30,
                 from socket.c:42:
/usr/include/strings.h:46:13: note: expected 'void *' but argument is of type 'int'
socket.c:1808:6: error: 'struct msghdr' has no member named 'msg_controllen'
gmake: *** [socket.o] Error 1

Well, that didn’t work as hoped, but after looking at the source and doing a quick search, I realized I could get it to build (and work, based on my limited testing), using:

CFLAGS="-D_XOPEN_SOURCE -D_XOPEN_SOURCE_EXTENDED=1 -D__EXTENSIONS__" ./configure --prefix=/usr/local

And then building with gmake, it works just fine. Looks like the project is alive again.

HBO Go on Linux

September 30th, 2013

HBO Go stopped working for me on Linux recently (actually it might not have been super recent, I don’t use it that often). Apparently a DRM problem with Flash 11.2.x on Gentoo (and other Linux versions I’m assuming), I was able to get it working again by emerging “media-libs/hal-flash” (version 0.2.0_rc1 I assume). I have no idea why HAL is “deprecated”.

The move away from Flash is going to eventually be OK, since it came with its own issues, particularly on 64 bit Linux. In the short term it has caused me a lot of problems

Setting up repository mirrors for local use

August 29th, 2013

I have a few OpenIndiana servers now, at home and at work. I’ve been working hard on clustering lately. To help with my work, in both places I’m busy setting up a local repository, something I can update periodically and then use it to update all my servers and containers. I’ve had a lot of success with a local http-replicator cache for Gentoo, and I hope this will prove useful as well. I’m not all that familiar with IPS anyway, so anything I can learn here would be useful. I sometimes miss the simplicity of SVR4 packages but they never added the features the other distributions were busy pioneering.

Based on some documentation I’ve been reading, to create the repository on my local machine, in a data zpool that I’d already created:

mkdir /data/pkg/dev
pkgrepo create /data/pkg/dev
pkgrecv -s -d /data/pkg/dev '*'
pkgrepo rebuild -s /data/pkg/dev

To set up the dev pkg.depotd server:

svccfg -s pkg/server add dev
svccfg -s pkg/server:dev addpg pkg application
svccfg -s pkg/server:dev setprop pkg/port=10081
svccfg -s pkg/server:dev setprop pkg/inst_root=/data/pkg/dev
svccfg -s pkg/server:dev addpg general framework
svccfg -s pkg/server:dev addpropvalue general/complete astring: dev
svccfg -s pkg/server:dev addpropvalue general/enabled boolean: true
svccfg -s pkg/server:dev setprop pkg/readonly=true
svccfg -s pkg/server:dev setprop pkg/threads=100

Some of these commands are borrowed from Solaris 11 instructions. I still need to look up some of these properties and find out if they are documented. Before today I had only seen the pkg/port, pkg/inst_root and pkg/readonly properties.

To enable it:

svcadm refresh application/pkg/server:dev
svcadm enable application/pkg/server:dev

After all the depot servers are up and running, I added this to the /etc/apache2/2.2/conf/proxy.conf file:

ProxyRequests Off
ProxyVia Block
ProxyStatus On
ProxyPreserveHost Off
ProxyPass /dev nocanon
ProxyPass /sfe nocanon
ProxyPass /sfe-encumbered nocanon
ProxyPass /local nocanon
ProxyPass /legacy nocanon
AllowEncodedSlashes NoDecode

After that is done, you need to set up the clients to use it. To replace the publisher:

pkg set-publisher -G '*' -g
pkg refresh --full

After that, I was able to do ‘pkg update’ as expected. I was able to mirror dev, sfe, sfe-encumbered and legacy without much trouble.

Now that I have that, I’ve created a local repository for development. This will be fun. I am wondering how to feed back some packages to the community. I should probably ask.

Tomcat 6.x latency on POST

July 22nd, 2013

We were having some problems with HTTP POST performance from the West coast to a tomcat server farm on the East coast. None of these problems showed up locally (between two hosts in the same datacenter.) The latency between West and East coast sites is on the order of 70ms. After much investigation, we (two of my coworkers and I) managed to reduce a post of a 32MB file from ~87 seconds to ~4 seconds with a combination of two things: changing socket.rxBufSize and changing /etc/sysctl.conf:

net.ipv4.tcp_mem=196608 262144 1048576
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 1048576 67108864
net.ipv4.tcp_wmem = 4096 1048576 67108864
net.core.netdev_max_backlog = 250000

In this case I think tcp_rmem and rmem_max are the important settings here. One of the odd things about this was the TCP window wasn’t scaling up…. the most I was seeing before this was 38K or so. Not at all what I expected.

By just changing the Tomcat 6 server.xml buffer size we got the time down from 87 to 39 seconds. A further change of the sysctl parameters got us down to just over 4 seconds. The changes we made to the sysctl parameters were limited to the second argument to net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Those other parameters were set for a previous unsuccessful attempt to improve the situation.

Also, these values were arrived at unscientifically. There are almost certainly some better choices out there. Based on some testing with stock CentOS 5.x scp, I think I should be able to get it to 2 seconds or better.

Pacemaker cluster on OpenIndiana

April 3rd, 2013

Today I finally got my NAS cluster working on the 3 OI 151a7 boxes I have set up in the lab at work for this purpose.

I took the work Mike Rowell had done on Linux-HA clustering and tweaked it a bit, and then added an OCF resource script to handle zpools and SCSI-3 persistent reservations groups. The idea of doing this without SCSI-3 PRG seemed like a disaster waiting to happen, but the sg3_utils that are available seem to work fine with our EMC Vplex.

I’ll have to document the details, perhaps in a future post here or something. It’s far from polished yet, but I can move zpools between nodes in the cluster with crm and crm_resource, and they’re exported as expected. Maybe it’ll help someone else trying to do HA SAN work with OI.

I’m hoping to make this work with OmniOS too.


January 19th, 2013

Looks like it’s finally time to migrate to sys-fs/eudev. udev 197 was unmasked today on amd64, and I would rather not go down that path.

At least I have an alternative, so I don’t have to go it alone.

I ended up having to use kmod, instead of module-init-tools. modprobe -l no longer works. I guess that feature is deprecated, and I’m supposed to use find in /lib/modules now. That seems less convenient. Other than that they seem the same.

ZFS assertion failed for SAN devices

January 8th, 2013

This was frustrating:

# zpool import nas2
Assertion failed: rn->rn_nozpool == B_FALSE, file ../common/libzfs_import.c, line 1086, function zpool_open_func

Discovered that I needed to tell zpool where to look for the devices:

zpool import -d /devices/scsi_vhci nas2

There was a time when I would have known that already, if we still used Solaris in any way at work. I wish the error message was more descriptive. Considering asking about it on the openindiana or illumos mailing lists.

Apache proxy relay OpenIndiana pkg install

May 22nd, 2012


This took me a bit of time to figure out. I needed to update some OpenIndiana boxes from 151a to 151a4 and they had no direct access to the internet. I did, however, have a box that I could relay the requests through. I first figured this method out, and then later I found a hint that there is a better way to do it. I don’t have the details on the “new” way written down, but I will include this other Apache virtual host proxy thing in case someone finds it useful.

I ended up adding this to the 00_default_vhost.conf file (on Gentoo, but a similar incantation would work on Apache 2.2 elsewhere I’m sure):

Listen 8080
NameVirtualHost *:8080

<virtualhost *:8080>
        Include /etc/apache2/vhosts.d/default_vhost.include

        <ifmodule mpm_peruser_module>
                ServerEnvironment apache apache
        ProxyRequests On
        ProxyVia Block
        ProxyStatus On
        ProxyPreserveHost Off
        ProxyPass /dev/
        ProxyPass /legacy/
        ProxyPass /sfe/
        ProxyPass /sfe-encumbered/
        ProxyPassReverse /dev/
        ProxyPassReverse /legacy/
        ProxyPassReverse /sfe/
        ProxyPassReverse /sfe-encumbered/
        AllowEncodedSlashes NoDecode


This was cobbled together from various sources. I wish I’d kept some references to them. The tricky bit for me was the AllowEncodedSlashes, an Apache directive I had never heard of before. Before that it can’t find packages because it translates %2F to /, and therefore the OpenIndiana pkg.depotd server can’t find the right file.

After that you just point at this with pkg set-publisher:

pfexec pkg set-publisher -p

You will also need to clean up the old publisher if I recall correctly.

Anyway, it kept me from having to copying all of OpenIndiana and setting up a repository locally.

Basic Linux firewall

May 22nd, 2012

Starting on a new Gentoo box, I was putting together a new firewall setup, and I thought I’d put my hacked down firewall setup script here so I’ll have something to start with next time. I used to try some of the other “higher level” tools to generate my firewall, but eventually they all got on my nerves. It was worth it, finally, to sit down for a couple of hours and understand what iptables does. In a lot of ways I prefer it to the Solaris ipf firewall tools now, but that is just personal preference, they are both very capable. I am hardly an expert on either one (or firewalls in general), but they can be useful tools, and provide some peace of mind. I also use this in conjunction with TCP wrappers (/etc/hosts.allow and /etc/hosts.deny).

EDIT: The original script has been modified below. They changed state tracking for established connections to use the “conntrack” module, and so I’ve updated what I use by default, and I’d like to not lead anyone astray if they find this old entry. Nor do I want to confuse myself the next time I need to do this.

Anyway, here is a very basic firewall setup script:


# Flush all the rules
/sbin/iptables -F

# Set the default policy for inbound/forwarded/outbound traffic.
/sbin/iptables -P INPUT DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -P OUTPUT ACCEPT

# Accept anything on loopback interface.
/sbin/iptables -A INPUT -i lo -j ACCEPT

# Accept traffic from this box to its own IP (e.g.
/sbin/iptables -A INPUT -s -j ACCEPT

# Allow state tracking.
#/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Accept incoming SSH connections.
# You may want to add some source (-s) addresses to this one, depending on
# your security policy.
/sbin/iptables -A INPUT -p tcp --dport ssh -j ACCEPT

# Accept incoming connections from to http/https.
/sbin/iptables -A INPUT -p tcp -s --dport http -j ACCEPT
/sbin/iptables -A INPUT -p tcp -s --dport https -j ACCEPT

# Display all your rules.
/sbin/iptables -L -v -n --line-numbers

# IPv6 example - most people should not need this today, but I use IPv6 networking
# internally just for fun.
/sbin/ip6tables -F

/sbin/ip6tables -P INPUT DROP
/sbin/ip6tables -P FORWARD DROP
/sbin/ip6tables -P OUTPUT ACCEPT

/sbin/ip6tables -A INPUT -i lo -s ::1/128 -j ACCEPT
# This address is specific to my host.  Get your own.  This prefix is for autoconfig anyway.
/sbin/ip6tables -A INPUT -s fe80::4a5b:39ff:fe67:9b7/128 -d fe80::4a5b:39ff:fe67:9b7/128 -j ACCEPT

#/sbin/ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/ip6tables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

/sbin/ip6tables -A INPUT -p tcp --dport ssh -j ACCEPT
/sbin/ip6tables -A INPUT -p tcp --dport http -j ACCEPT
/sbin/ip6tables -A INPUT -p tcp --dport https -j ACCEPT

/sbin/ip6tables -A INPUT -p ipv6-icmp -j ACCEPT

/sbin/ip6tables -L -v -n --line-numbers

After you run the script, the rules will be installed. You have to be careful if you’re doing this on a box you can’t get into via other means (iLO, DRAC, physical console). When testing remotely I sometimes run this with a script in cron to clear all the rules.

A cron entry like this will reset the rules on the quarter hour, in case you get locked out:

0,15,30,45 * * * * /sbin/iptables-restore < /root/firewall_reset

And /root/firewall_reset contains:

:INPUT ACCEPT [164:15203]
:OUTPUT ACCEPT [147:63028]

:INPUT ACCEPT [164:15203]
:OUTPUT ACCEPT [147:63028]


One you are satisfied with your firewall, you can save the rules with:

/etc/init.d/iptables save
/etc/init.d/ip6tables save

Obviously, make sure you disable the cron job above.