PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic Segmentation

Mingyang Jiang, Yiran Wu, Tianqi Zhao, Zelin Zhao, Cewu Lu*

(* corresponding author: lucewu@sjtu.edu.cn)


Recently, 3D understanding research pays more attention to extracting the feature from point cloud directly. Therefore, exploring shape pattern description in points is essential. Inspired by SIFT that is an outstanding 2D shape representation, we design a PointSIFT module that encodes information of different orientations and is adaptive to scale of shape. Specifically, an orientation-encoding unit is designed to describe eight crucial orientations. Thus, by stacking several orientation-encoding units, we can get the multi-scale representation. Extensive experiments show our PointSIFT-based framework outperform state-of-the-art method on standard benchmarking datasets.
Figure 1. Illustration of the details in Orientation-encoding Point Convolutional layer. (a): point clouds in 3D space. (b) neighbors in eight directions. (c) three stages convolution combines all the features. Figure 2. Illustration of end-to-end network architecture. The network contains two part: Encoder (Set Abstraction) and Decoder (Feature Propagation). PointSIFT modules (red color) are inserted between all adjacent layers of Set Abstraction (blue color) and Feature Propagation module (green color). Both set Abstraction module and Feature Propagation module are introduced in PointNet paper. The encoder and the decoder are linked by several skip connections.

Code and Paper

Our preprint paper is here. Our main module, training and evaluating code is available on Github.
You can simply use it in your project by:
import tensorflow as tf
# import our module
from tf_utils.pointSIFT_util import pointSIFT_module
# input coordinates
xyz = tf.tf.placeholder(tf.float32, shape=(batch_size, num_point, 3))
# input features
point_feature = tf.tf.placeholder(tf.float32, shape=(batch_size, num_point, input_dim)
# setting phases
is_training = tf.placeholder(dtype=tf.bool, shape=())
# setting searching radius (0.1 as an example)
radius = 0.1
_, out_feature, _ = pointSIFT_module(xyz, point_feature, radius, output_dim, is_training)

Example Results